Introduction

Recall, the testing set of a model cannot be used during the training process; it is meant to simulate how well a model would perform using unseen data. That means that we need to find a way to evaluate model performance before making predictions using the testing set. This chapter focuses on how we accomplish this task using resampling.

Resubstitution

In the previous two chapters, we worked with the predictions from training a linear regression model. Now we’ll build another regression model using random forest.

# Fit 
(rf_fit <- fit(rf_workflow, ames_train))
══ Workflow [trained] ══════════════════════════════════════════════════
Preprocessor: Formula
Model: rand_forest()

── Preprocessor ────────────────────────────────────────────────────────
sale_price ~ neighborhood + gr_liv_area + year_built + bldg_type + 
    latitude + longitude

── Model ───────────────────────────────────────────────────────────────
Ranger result

Call:
 ranger::ranger(x = maybe_data_frame(x), y = y, num.trees = ~1000,      num.threads = 1, verbose = FALSE, seed = sample.int(10^5,          1)) 

Type:                             Regression 
Number of trees:                  1000 
Sample size:                      2342 
Number of independent variables:  6 
Mtry:                             2 
Target node size:                 5 
Variable importance mode:         none 
Splitrule:                        variance 
OOB prediction error (MSE):       0.005315515 
R squared (OOB):                  0.82922 

Now we’ll predict the training data (note, this is just used for demonstration to compare the two models).

Based on the resubstitution error rate, the random forest model is better at predicting the training data compared to the ordinary least squares regression. Now we’ll evaluate how well the random forest model performs on the test data.

Now we see that the RMSE is much higher and the coefficient of determination is lower- this is because the model did not generalize as well to new unseen data. In this context, the random forest model has low bias, but higher variance. For thoroughness, we can compute the same statistics for the ordinary least squares model on the test data.

The RMSE and coefficient of determination are almost the same between the training and test sets- this is because although not as accurate on the training set (i.e., higher bias), ordinary least squares models tend to have lower variance compared to more flexible models like random forest.

Resampling methods

Cross-validation

Resampling methods further split the training data into analysis sets used to train and tune the model and assessment sets used to evaluate model performance. The latter are almost like pseudo testing sets, but ensure that no data leakage occurs from the testing set into the model training process.

One resampling method is cross-validation, called V-fold cross-validation in the textbook. For each iteration, the data set is randomly split into V folds of relatively equal size. V-1 folds are used for training and 1 fold is retained for assessment. V iterations are performed such that for each iteration, a different fold is used as the holdout set, and the performance statistics are averaged over the V folds. In general, 10-fold cross-validation is used, which is what we will use here. In general, as the value of V decreases, bias increases and variance decreases; however, with larger values of V, bias decreases at the expense of an increase in variance.

(ames_folds <- vfold_cv(ames_train, v = 10))
#  10-fold cross-validation 

The Central Limit Theorem states that with repeated sampling, summary statistics will converge toward a normal distribution. We can simulate sampling more data by performing repeat cross-validation, essentially by performing the V-fold cross-validation just described, repeated R times. We can easily simulate repeated V-fold cross-validation using the repeats argument in the vfold_cv() call.

vfold_cv(ames_train, v = 10, repeats = 5)
#  10-fold cross-validation repeated 5 times 

A slightly different flavor of V-fold cross-validation is Monte Carlo cross-validation, which randomly allocates a proportion of the data into assessment sets for each iteration. Now we’ll create a validation set, which is a partition of the training set.

(val_set <- validation_split(ames_train, prop = 0.7))
# Validation Set Split (0.7/0.3)  

Bootstrapping

Another resampling method is the bootstrap, which entails redrawing samples at random from the original data set, with replacement, to create a new data set of the same size. The analysis set is the new bootstrap data set; the assessment set, called the “out of bag,” is composed of the training set samples that were not resampled in the analysis set. Importantly, because the asssessment sets are composed of samples that were not resampled in the analysis set, and samples can be drawn more than once in the analysis set, the sizes of the assessment sets may vary.

In general, bootstrapping results in performance estimates with low variance, but higher bias.

bootstraps(ames_train, times = 10)
# Bootstrap sampling 

Estimating performance

All of the metrics discussed previously may be used to evaluate model performance in the assessment sets. By default, the RMSE and coefficient of determination are the output metrics for regression models, while AUC and accuracy are output metrics for classification models. Below we’ll fit the random forest worfklow using 10-fold cross-validation and keep the predictions for each iteration of cross-validation.

This is a helpful strategy for identifying predictions that were way off from the model. Looking at the scatterplot, there appear to be one or two houses that have a low sale price but were overpredicted by the model. We can find out which ones these are by looking at the the residuals, or the difference between the actual sale price and the predicted value.

The results from the resampling are correlated with the results from the validation set. In the textbook, the authors recommend repeating the process and setting a different seed for the pseudo random number generation. After setting the seet from 55 to 123, the results are largely the same, demonstrating the power of resampling methods to estimate how well a model may perform when it sees new data.

Parallel processing

Since each resampled data set is independent of the next, the model training process can be parallelized using the doMC package. The fit_resamples() function requires that the user register parallel cores to be used before fitting the models.

Saving resampled objects

If desired, the models fitted to resampled data sets can be saved and extracted. This could be interesting to see how the model coefficients and other metrics change for each resample, but by default, these typically aren’t retained because they’re just being used to evaluate performance.

# Save folds, predictions, and random forest results
LS0tCnRpdGxlOiAiQ2gxMDogUmVzYW1wbGluZyIKb3V0cHV0OgogIGh0bWxfbm90ZWJvb2s6CiAgICB0b2M6IHRydWUKICAgIHRvY19mbG9hdDogdHJ1ZQogICAgdGhlbWU6IGNlcnVsZWFuCi0tLQoKIyMgSW50cm9kdWN0aW9uCgpSZWNhbGwsIHRoZSB0ZXN0aW5nIHNldCBvZiBhIG1vZGVsIGNhbm5vdCBiZSB1c2VkIGR1cmluZyB0aGUgdHJhaW5pbmcgcHJvY2VzczsgaXQgaXMgbWVhbnQgdG8gc2ltdWxhdGUgaG93IHdlbGwgYSBtb2RlbCB3b3VsZCBwZXJmb3JtIHVzaW5nIHVuc2VlbiBkYXRhLiAgVGhhdCBtZWFucyB0aGF0IHdlIG5lZWQgdG8gZmluZCBhIHdheSB0byBldmFsdWF0ZSBtb2RlbCBwZXJmb3JtYW5jZSBiZWZvcmUgbWFraW5nIHByZWRpY3Rpb25zIHVzaW5nIHRoZSB0ZXN0aW5nIHNldC4gIFRoaXMgY2hhcHRlciBmb2N1c2VzIG9uIGhvdyB3ZSBhY2NvbXBsaXNoIHRoaXMgdGFzayB1c2luZyByZXNhbXBsaW5nLiAgCgpgYGB7ciBub3RlYm9vayBzZXR1cH0KIyBMaWJyYXJpZXMgCmxpYnJhcnkocGFjbWFuKQpwX2xvYWQodGlkeXZlcnNlLCB0aWR5bW9kZWxzLCBzcGxpbmVzLCBicm9vbSwgamFuaXRvcikKCiMgTG9hZCBkYXRhCmRhdGEoYW1lcywgcGFja2FnZSA9ICJtb2RlbGRhdGEiKQphbWVzIDwtIGFtZXMgJT4lIAogIGNsZWFuX25hbWVzKCkgJT4lIAogIG11dGF0ZShzYWxlX3ByaWNlID0gbG9nMTAoc2FsZV9wcmljZSkpCgojIFNwbGl0CnNldC5zZWVkKDEyMykKYW1lc19zcGxpdCA8LSBpbml0aWFsX3NwbGl0KGFtZXMsIHByb3AgPSAwLjgsIHN0cmF0YSA9IHNhbGVfcHJpY2UpCmFtZXNfdHJhaW4gPC0gdHJhaW5pbmcoYW1lc19zcGxpdCkKYW1lc190ZXN0IDwtIHRlc3RpbmcoYW1lc19zcGxpdCkKCiMgTG9hZCByZWNpcGUsIHdvcmtmbG93LCBhbmQgbW9kZWwgZml0CmxtX3dvcmtmbG93IDwtIHJlYWRSRFMoIi4uL2RhdGEvYW1lc19sbV93b3JrZmxvdy5SZHMiKQphbWVzX3JlY2lwZSA8LSByZWFkUkRTKCIuLi9kYXRhL2FtZXNfcmVjaXBlLlJkcyIpCmxtX2ZpdCA8LSByZWFkUkRTKCIuLi9kYXRhL2FtZXNfbG1fZml0LlJkcyIpCmBgYAoKIyMgUmVzdWJzdGl0dXRpb24gCgpJbiB0aGUgcHJldmlvdXMgdHdvIGNoYXB0ZXJzLCB3ZSB3b3JrZWQgd2l0aCB0aGUgcHJlZGljdGlvbnMgZnJvbSB0cmFpbmluZyBhIGxpbmVhciByZWdyZXNzaW9uIG1vZGVsLiAgTm93IHdlJ2xsIGJ1aWxkIGFub3RoZXIgcmVncmVzc2lvbiBtb2RlbCB1c2luZyByYW5kb20gZm9yZXN0LgoKYGBge3IgcmFuZG9tIGZvcmVzdH0KIyBEZWZpbmUgbW9kZWwgc3BlY2lmaWNhdGlvbgpyZl9tb2QgPC0gcmFuZF9mb3Jlc3QodHJlZXMgPSAxMDAwKSAlPiUgCiAgc2V0X2VuZ2luZSgicmFuZ2VyIikgJT4lIAogIHNldF9tb2RlKCJyZWdyZXNzaW9uIikKCiMgQnVpbGQgd29ya2Zsb3cKKHJmX3dvcmtmbG93IDwtIHdvcmtmbG93KCkgJT4lIAogIGFkZF9mb3JtdWxhKHNhbGVfcHJpY2UgfiBuZWlnaGJvcmhvb2QgKyBncl9saXZfYXJlYSArIHllYXJfYnVpbHQgKyAKICAgICAgICAgICAgICAgIGJsZGdfdHlwZSArIGxhdGl0dWRlICsgbG9uZ2l0dWRlKSAlPiUgCiAgYWRkX21vZGVsKHJmX21vZCkpCgojIEZpdCAKKHJmX2ZpdCA8LSBmaXQocmZfd29ya2Zsb3csIGFtZXNfdHJhaW4pKQpgYGAKCk5vdyB3ZSdsbCBwcmVkaWN0IHRoZSB0cmFpbmluZyBkYXRhIChub3RlLCB0aGlzIGlzIGp1c3QgdXNlZCBmb3IgZGVtb25zdHJhdGlvbiB0byBjb21wYXJlIHRoZSB0d28gbW9kZWxzKS4gIAoKYGBge3IgcmVzdWJzdGl0dXRpb24gZXJyb3IgcmF0ZX0KIyBCdWlsZCBmdW5jdGlvbiB0byBwcmVkaWN0IHRoZSB0cmFpbmluZyBkYXRhCmVzdGltYXRlX3BlcmYgPC0gZnVuY3Rpb24obW9kZWwsIGRhdCkgewogIAogIGNsIDwtIG1hdGNoLmNhbGwoKQogIG9ial9uYW1lIDwtIGFzLmNoYXJhY3RlcihjbCRtb2RlbCkKICBkYXRhX25hbWUgPC0gYXMuY2hhcmFjdGVyKGNsJGRhdCkKICBkYXRhX25hbWUgPC0gZ3N1YigiYW1lc18iLCAiIiwgZGF0YV9uYW1lKQogIAogICMgRXN0aW1hdGUgbWV0cmljIHNldAogIHJlZ19tZXRyaWNzIDwtIG1ldHJpY19zZXQocm1zZSwgcnNxKQogIAogICMgUHJlZGljdCB0cmFpbmluZyBkYXRhIGFuZCB0aWR5IHNlbGVjdGVkIG1ldHJpY3MKICBvdXQgPC0gbW9kZWwgJT4lIAogICAgcHJlZGljdChkYXQpICU+JSAKICAgIGJpbmRfY29scyhkYXQgJT4lIHNlbGVjdChzYWxlX3ByaWNlKSkgJT4lIAogICAgcmVnX21ldHJpY3ModHJ1dGggPSBzYWxlX3ByaWNlLCAucHJlZCkgJT4lIAogICAgc2VsZWN0KC0uZXN0aW1hdG9yKSAlPiUgCiAgICBtdXRhdGUob2JqZWN0ID0gb2JqX25hbWUsIGRhdGEgPSBkYXRhX25hbWUpCiAgCiAgcmV0dXJuKG91dCkKCn0KCmVzdGltYXRlX3BlcmYocmZfZml0LCBhbWVzX3RyYWluKSAlPiUgCiAgYmluZF9yb3dzKGVzdGltYXRlX3BlcmYobG1fZml0LCBhbWVzX3RyYWluKSkgJT4lIAogIGFycmFuZ2UoLm1ldHJpYykKCmBgYAoKQmFzZWQgb24gdGhlIHJlc3Vic3RpdHV0aW9uIGVycm9yIHJhdGUsIHRoZSByYW5kb20gZm9yZXN0IG1vZGVsIGlzIGJldHRlciBhdCBwcmVkaWN0aW5nIHRoZSB0cmFpbmluZyBkYXRhIGNvbXBhcmVkIHRvIHRoZSBvcmRpbmFyeSBsZWFzdCBzcXVhcmVzIHJlZ3Jlc3Npb24uICBOb3cgd2UnbGwgZXZhbHVhdGUgaG93IHdlbGwgdGhlIHJhbmRvbSBmb3Jlc3QgbW9kZWwgcGVyZm9ybXMgb24gdGhlIHRlc3QgZGF0YS4KCmBgYHtyIHJhbmRvbSBmb3Jlc3QgdGVzdCBkYXRhfQplc3RpbWF0ZV9wZXJmKHJmX2ZpdCwgYW1lc190ZXN0KQpgYGAKCk5vdyB3ZSBzZWUgdGhhdCB0aGUgUk1TRSBpcyBtdWNoIGhpZ2hlciBhbmQgdGhlIGNvZWZmaWNpZW50IG9mIGRldGVybWluYXRpb24gaXMgbG93ZXItIHRoaXMgaXMgYmVjYXVzZSB0aGUgbW9kZWwgZGlkIG5vdCBnZW5lcmFsaXplIGFzIHdlbGwgdG8gbmV3IHVuc2VlbiBkYXRhLiAgSW4gdGhpcyBjb250ZXh0LCB0aGUgcmFuZG9tIGZvcmVzdCBtb2RlbCBoYXMgbG93IGJpYXMsIGJ1dCBoaWdoZXIgdmFyaWFuY2UuICBGb3IgdGhvcm91Z2huZXNzLCB3ZSBjYW4gY29tcHV0ZSB0aGUgc2FtZSBzdGF0aXN0aWNzIGZvciB0aGUgb3JkaW5hcnkgbGVhc3Qgc3F1YXJlcyBtb2RlbCBvbiB0aGUgdGVzdCBkYXRhLgoKYGBge3IgbGluZWFyIG1vZGVsIHRlc3QgZGF0YX0KZXN0aW1hdGVfcGVyZihsbV9maXQsIGFtZXNfdGVzdCkKCmBgYAoKVGhlIFJNU0UgYW5kIGNvZWZmaWNpZW50IG9mIGRldGVybWluYXRpb24gYXJlIGFsbW9zdCB0aGUgc2FtZSBiZXR3ZWVuIHRoZSB0cmFpbmluZyBhbmQgdGVzdCBzZXRzLSB0aGlzIGlzIGJlY2F1c2UgYWx0aG91Z2ggbm90IGFzIGFjY3VyYXRlIG9uIHRoZSB0cmFpbmluZyBzZXQgKGkuZS4sIGhpZ2hlciBiaWFzKSwgb3JkaW5hcnkgbGVhc3Qgc3F1YXJlcyBtb2RlbHMgdGVuZCB0byBoYXZlIGxvd2VyIHZhcmlhbmNlIGNvbXBhcmVkIHRvIG1vcmUgZmxleGlibGUgbW9kZWxzIGxpa2UgcmFuZG9tIGZvcmVzdC4gIAoKIyMgUmVzYW1wbGluZyBtZXRob2RzCgojIyMgQ3Jvc3MtdmFsaWRhdGlvbgoKUmVzYW1wbGluZyBtZXRob2RzIGZ1cnRoZXIgc3BsaXQgdGhlIHRyYWluaW5nIGRhdGEgaW50byBhbmFseXNpcyBzZXRzIHVzZWQgdG8gdHJhaW4gYW5kIHR1bmUgdGhlIG1vZGVsIGFuZCBhc3Nlc3NtZW50IHNldHMgdXNlZCB0byBldmFsdWF0ZSBtb2RlbCBwZXJmb3JtYW5jZS4gIFRoZSBsYXR0ZXIgYXJlIGFsbW9zdCBsaWtlIHBzZXVkbyB0ZXN0aW5nIHNldHMsIGJ1dCBlbnN1cmUgdGhhdCBubyBkYXRhIGxlYWthZ2Ugb2NjdXJzIGZyb20gdGhlIHRlc3Rpbmcgc2V0IGludG8gdGhlIG1vZGVsIHRyYWluaW5nIHByb2Nlc3MuCgpPbmUgcmVzYW1wbGluZyBtZXRob2QgaXMgY3Jvc3MtdmFsaWRhdGlvbiwgY2FsbGVkIFYtZm9sZCBjcm9zcy12YWxpZGF0aW9uIGluIHRoZSB0ZXh0Ym9vay4gIEZvciBlYWNoIGl0ZXJhdGlvbiwgdGhlIGRhdGEgc2V0IGlzIHJhbmRvbWx5IHNwbGl0IGludG8gViBmb2xkcyBvZiByZWxhdGl2ZWx5IGVxdWFsIHNpemUuICBWLTEgZm9sZHMgYXJlIHVzZWQgZm9yIHRyYWluaW5nIGFuZCAxIGZvbGQgaXMgcmV0YWluZWQgZm9yIGFzc2Vzc21lbnQuICBWIGl0ZXJhdGlvbnMgYXJlIHBlcmZvcm1lZCBzdWNoIHRoYXQgZm9yIGVhY2ggaXRlcmF0aW9uLCBhIGRpZmZlcmVudCBmb2xkIGlzIHVzZWQgYXMgdGhlIGhvbGRvdXQgc2V0LCBhbmQgdGhlIHBlcmZvcm1hbmNlIHN0YXRpc3RpY3MgYXJlIGF2ZXJhZ2VkIG92ZXIgdGhlIFYgZm9sZHMuICBJbiBnZW5lcmFsLCAxMC1mb2xkIGNyb3NzLXZhbGlkYXRpb24gaXMgdXNlZCwgd2hpY2ggaXMgd2hhdCB3ZSB3aWxsIHVzZSBoZXJlLiAgSW4gZ2VuZXJhbCwgYXMgdGhlIHZhbHVlIG9mIFYgZGVjcmVhc2VzLCBiaWFzIGluY3JlYXNlcyBhbmQgdmFyaWFuY2UgZGVjcmVhc2VzOyBob3dldmVyLCB3aXRoIGxhcmdlciB2YWx1ZXMgb2YgViwgYmlhcyBkZWNyZWFzZXMgYXQgdGhlIGV4cGVuc2Ugb2YgYW4gaW5jcmVhc2UgaW4gdmFyaWFuY2UuICAKCmBgYHtyIGNyb3NzLXZhbGlkYXRpb259CiMgRGVmaW5lIGZvbGRzCnNldC5zZWVkKDEyMykKKGFtZXNfZm9sZHMgPC0gdmZvbGRfY3YoYW1lc190cmFpbiwgdiA9IDEwKSkKCiMgRWFjaCBzcGxpdCBlbGVtZW50IHNob3dzIHRoZSBzYW1wbGUgc2l6ZSBmb3IgdGhlIGFuYWx5c2lzIGFuZCBhc3Nlc3NtZW50IHNldHMKYW1lc19mb2xkcyRzcGxpdHNbWzFdXSAlPiUgCiAgYW5hbHlzaXMoKSAlPiUgCiAgZGltKCkKYGBgCgpUaGUgQ2VudHJhbCBMaW1pdCBUaGVvcmVtIHN0YXRlcyB0aGF0IHdpdGggcmVwZWF0ZWQgc2FtcGxpbmcsIHN1bW1hcnkgc3RhdGlzdGljcyB3aWxsIGNvbnZlcmdlIHRvd2FyZCBhIG5vcm1hbCBkaXN0cmlidXRpb24uICBXZSBjYW4gc2ltdWxhdGUgc2FtcGxpbmcgbW9yZSBkYXRhIGJ5IHBlcmZvcm1pbmcgcmVwZWF0IGNyb3NzLXZhbGlkYXRpb24sIGVzc2VudGlhbGx5IGJ5IHBlcmZvcm1pbmcgdGhlIFYtZm9sZCBjcm9zcy12YWxpZGF0aW9uIGp1c3QgZGVzY3JpYmVkLCByZXBlYXRlZCBSIHRpbWVzLiAgV2UgY2FuIGVhc2lseSBzaW11bGF0ZSByZXBlYXRlZCBWLWZvbGQgY3Jvc3MtdmFsaWRhdGlvbiB1c2luZyB0aGUgYHJlcGVhdHNgIGFyZ3VtZW50IGluIHRoZSBgdmZvbGRfY3YoKWAgY2FsbC4KCmBgYHtyIHJlcGVhdGVkIGNyb3NzLXZhbGlkYXRpb259CnZmb2xkX2N2KGFtZXNfdHJhaW4sIHYgPSAxMCwgcmVwZWF0cyA9IDUpCgpgYGAKCkEgc2xpZ2h0bHkgZGlmZmVyZW50IGZsYXZvciBvZiBWLWZvbGQgY3Jvc3MtdmFsaWRhdGlvbiBpcyBNb250ZSBDYXJsbyBjcm9zcy12YWxpZGF0aW9uLCB3aGljaCByYW5kb21seSBhbGxvY2F0ZXMgYSBwcm9wb3J0aW9uIG9mIHRoZSBkYXRhIGludG8gYXNzZXNzbWVudCBzZXRzIGZvciBlYWNoIGl0ZXJhdGlvbi4gIE5vdyB3ZSdsbCBjcmVhdGUgYSB2YWxpZGF0aW9uIHNldCwgd2hpY2ggaXMgYSBwYXJ0aXRpb24gb2YgdGhlIHRyYWluaW5nIHNldC4gIAoKYGBge3IgdmFsaWRhdGlvbiBzZXR9CnNldC5zZWVkKDEyMykKKHZhbF9zZXQgPC0gdmFsaWRhdGlvbl9zcGxpdChhbWVzX3RyYWluLCBwcm9wID0gMC43KSkKCmBgYAoKIyMjIEJvb3RzdHJhcHBpbmcKCkFub3RoZXIgcmVzYW1wbGluZyBtZXRob2QgaXMgdGhlIGJvb3RzdHJhcCwgd2hpY2ggZW50YWlscyByZWRyYXdpbmcgc2FtcGxlcyBhdCByYW5kb20gZnJvbSB0aGUgb3JpZ2luYWwgZGF0YSBzZXQsIHdpdGggcmVwbGFjZW1lbnQsIHRvIGNyZWF0ZSBhIG5ldyBkYXRhIHNldCBvZiB0aGUgc2FtZSBzaXplLiAgVGhlIGFuYWx5c2lzIHNldCBpcyB0aGUgbmV3IGJvb3RzdHJhcCBkYXRhIHNldDsgdGhlIGFzc2Vzc21lbnQgc2V0LCBjYWxsZWQgdGhlICJvdXQgb2YgYmFnLCIgaXMgY29tcG9zZWQgb2YgdGhlIHRyYWluaW5nIHNldCBzYW1wbGVzIHRoYXQgd2VyZSBub3QgcmVzYW1wbGVkIGluIHRoZSBhbmFseXNpcyBzZXQuICBJbXBvcnRhbnRseSwgYmVjYXVzZSB0aGUgYXNzc2Vzc21lbnQgc2V0cyBhcmUgY29tcG9zZWQgb2Ygc2FtcGxlcyB0aGF0IHdlcmUgbm90IHJlc2FtcGxlZCBpbiB0aGUgYW5hbHlzaXMgc2V0LCBhbmQgc2FtcGxlcyBjYW4gYmUgZHJhd24gbW9yZSB0aGFuIG9uY2UgaW4gdGhlIGFuYWx5c2lzIHNldCwgdGhlIHNpemVzIG9mIHRoZSBhc3Nlc3NtZW50IHNldHMgbWF5IHZhcnkuIAoKSW4gZ2VuZXJhbCwgYm9vdHN0cmFwcGluZyByZXN1bHRzIGluIHBlcmZvcm1hbmNlIGVzdGltYXRlcyB3aXRoIGxvdyB2YXJpYW5jZSwgYnV0IGhpZ2hlciBiaWFzLiAKCmBgYHtyIGNyZWF0ZSBib290c3RyYXAgZGF0YSBzZXR9CmJvb3RzdHJhcHMoYW1lc190cmFpbiwgdGltZXMgPSAxMCkKYGBgCgojIyBFc3RpbWF0aW5nIHBlcmZvcm1hbmNlCgpBbGwgb2YgdGhlIG1ldHJpY3MgZGlzY3Vzc2VkIHByZXZpb3VzbHkgbWF5IGJlIHVzZWQgdG8gZXZhbHVhdGUgbW9kZWwgcGVyZm9ybWFuY2UgaW4gdGhlIGFzc2Vzc21lbnQgc2V0cy4gIEJ5IGRlZmF1bHQsIHRoZSBSTVNFIGFuZCBjb2VmZmljaWVudCBvZiBkZXRlcm1pbmF0aW9uIGFyZSB0aGUgb3V0cHV0IG1ldHJpY3MgZm9yIHJlZ3Jlc3Npb24gbW9kZWxzLCB3aGlsZSBBVUMgYW5kIGFjY3VyYWN5IGFyZSBvdXRwdXQgbWV0cmljcyBmb3IgY2xhc3NpZmljYXRpb24gbW9kZWxzLiAgQmVsb3cgd2UnbGwgZml0IHRoZSByYW5kb20gZm9yZXN0IHdvcmZrbG93IHVzaW5nIDEwLWZvbGQgY3Jvc3MtdmFsaWRhdGlvbiBhbmQga2VlcCB0aGUgcHJlZGljdGlvbnMgZm9yIGVhY2ggaXRlcmF0aW9uIG9mIGNyb3NzLXZhbGlkYXRpb24uCgpgYGB7ciBjcm9zcy12YWxpYXRpb24gcmFuZG9tIGZvcmVzdH0KIyBTZXQgY29udHJvbCB0byBrZWVwIGFsbCBwcmVkaWN0aW9ucwprZWVwX3ByZWQgPC0gY29udHJvbF9yZXNhbXBsZXMoc2F2ZV9wcmVkID0gVFJVRSwgc2F2ZV93b3JrZmxvdyA9IFRSVUUpCgojIEZpdCByZXNhbXBsZXMKc2V0LnNlZWQoMTMwKQoocmZfcmVzIDwtIHJmX3dvcmtmbG93ICU+JSAKICBmaXRfcmVzYW1wbGVzKHJlc2FtcGxlcyA9IGFtZXNfZm9sZHMsIGNvbnRyb2wgPSBrZWVwX3ByZWQpKQoKIyBMb29rIGF0IHRoZSBtZWFuIHBlcmZvcm1hbmNlIG1ldHJpY3MgYWNyb3NzIGZvbGRzCmNvbGxlY3RfbWV0cmljcyhyZl9yZXMsIHN1bW1hcml6ZSA9IFRSVUUpCmBgYAoKYGBge3IgY29sbGVjdCBwcmVkaWN0aW9uc30KKGFzc2Vzc21lbnRfcmVzIDwtIGNvbGxlY3RfcHJlZGljdGlvbnMocmZfcmVzKSkKCiMgUGxvdCBhZ2FpbnN0IHRydWUgb3V0Y29tZQpnZ3Bsb3QoYXNzZXNzbWVudF9yZXMsIGFlcyhzYWxlX3ByaWNlLCAucHJlZCkpICsgCiAgZ2VvbV9hYmxpbmUobGluZXR5cGUgPSAiZGFzaGVkIiwgY29sb3IgPSAicmVkIikgKyAKICBnZW9tX3BvaW50KGFscGhhID0gMC4yKSArIAogIGNvb3JkX29ic19wcmVkKCkgKyAKICBsYWJzKHggPSAiU2FsZSBwcmljZSAobG9nMTApIiwgCiAgICAgICB5ID0gIlByZWRpY3RlZCBzYWxlIHByaWNlIChsb2cxMCkiKQoKCmBgYAoKVGhpcyBpcyBhIGhlbHBmdWwgc3RyYXRlZ3kgZm9yIGlkZW50aWZ5aW5nIHByZWRpY3Rpb25zIHRoYXQgd2VyZSB3YXkgb2ZmIGZyb20gdGhlIG1vZGVsLiAgTG9va2luZyBhdCB0aGUgc2NhdHRlcnBsb3QsIHRoZXJlIGFwcGVhciB0byBiZSBvbmUgb3IgdHdvIGhvdXNlcyB0aGF0IGhhdmUgYSBsb3cgc2FsZSBwcmljZSBidXQgd2VyZSBvdmVycHJlZGljdGVkIGJ5IHRoZSBtb2RlbC4gIFdlIGNhbiBmaW5kIG91dCB3aGljaCBvbmVzIHRoZXNlIGFyZSBieSBsb29raW5nIGF0IHRoZSB0aGUgcmVzaWR1YWxzLCBvciB0aGUgZGlmZmVyZW5jZSBiZXR3ZWVuIHRoZSBhY3R1YWwgc2FsZSBwcmljZSBhbmQgdGhlIHByZWRpY3RlZCB2YWx1ZS4KCmBgYHtyIGZpbmQgcG9vcmx5IHByZWRpY3RlZCBob21lc30KIyBJc29sYXRlIG92ZXItcHJlZGljdGVkIGRhdGEgCihvdmVyX3ByZWRpY3RlZCA8LSBhc3Nlc3NtZW50X3JlcyAlPiUgCiAgbXV0YXRlKHJlc2lkdWFsID0gc2FsZV9wcmljZSAtIC5wcmVkKSAlPiUgCiAgYXJyYW5nZShkZXNjKGFicyhyZXNpZHVhbCkpKSAlPiUgCiAgc2xpY2UoMToyKSkKCiMgTG9vayBhdCB2YXJpYWJsZXMgZnJvbSBvcmlnaW5hbCBkYXRhIHNldAphbWVzX3RyYWluICU+JSAKICBzbGljZShvdmVyX3ByZWRpY3RlZCQucm93KSAlPiUgCiAgc2VsZWN0KGdyX2xpdl9hcmVhLCBuZWlnaGJvcmhvb2QsIHllYXJfYnVpbHQsIGJlZHJvb21fYWJ2X2dyLCBmdWxsX2JhdGgpCmBgYApgYGB7ciB1c2UgdmFsaWRhdGlvbiBzZXR9Cih2YWxfcmVzIDwtIHJmX3dvcmtmbG93ICU+JSAKICBmaXRfcmVzYW1wbGVzKHJlc2FtcGxlcyA9IHZhbF9zZXQpKQoKY29sbGVjdF9tZXRyaWNzKHZhbF9yZXMpCmBgYAoKVGhlIHJlc3VsdHMgZnJvbSB0aGUgcmVzYW1wbGluZyBhcmUgY29ycmVsYXRlZCB3aXRoIHRoZSByZXN1bHRzIGZyb20gdGhlIHZhbGlkYXRpb24gc2V0LiAgSW4gdGhlIHRleHRib29rLCB0aGUgYXV0aG9ycyByZWNvbW1lbmQgcmVwZWF0aW5nIHRoZSBwcm9jZXNzIGFuZCBzZXR0aW5nIGEgZGlmZmVyZW50IHNlZWQgZm9yIHRoZSBwc2V1ZG8gcmFuZG9tIG51bWJlciBnZW5lcmF0aW9uLiAgQWZ0ZXIgc2V0dGluZyB0aGUgc2VldCBmcm9tIGA1NWAgdG8gYDEyM2AsIHRoZSByZXN1bHRzIGFyZSBsYXJnZWx5IHRoZSBzYW1lLCBkZW1vbnN0cmF0aW5nIHRoZSBwb3dlciBvZiByZXNhbXBsaW5nIG1ldGhvZHMgdG8gZXN0aW1hdGUgaG93IHdlbGwgYSBtb2RlbCBtYXkgcGVyZm9ybSB3aGVuIGl0IHNlZXMgbmV3IGRhdGEuCgojIyBQYXJhbGxlbCBwcm9jZXNzaW5nCgpTaW5jZSBlYWNoIHJlc2FtcGxlZCBkYXRhIHNldCBpcyBpbmRlcGVuZGVudCBvZiB0aGUgbmV4dCwgdGhlIG1vZGVsIHRyYWluaW5nIHByb2Nlc3MgY2FuIGJlIHBhcmFsbGVsaXplZCB1c2luZyB0aGUgYGRvTUNgIHBhY2thZ2UuICBUaGUgYGZpdF9yZXNhbXBsZXMoKWAgZnVuY3Rpb24gcmVxdWlyZXMgdGhhdCB0aGUgdXNlciByZWdpc3RlciBwYXJhbGxlbCBjb3JlcyB0byBiZSB1c2VkIGJlZm9yZSBmaXR0aW5nIHRoZSBtb2RlbHMuICAKCiMjIFNhdmluZyByZXNhbXBsZWQgb2JqZWN0cwoKSWYgZGVzaXJlZCwgdGhlIG1vZGVscyBmaXR0ZWQgdG8gcmVzYW1wbGVkIGRhdGEgc2V0cyBjYW4gYmUgc2F2ZWQgYW5kIGV4dHJhY3RlZC4gIFRoaXMgY291bGQgYmUgaW50ZXJlc3RpbmcgdG8gc2VlIGhvdyB0aGUgbW9kZWwgY29lZmZpY2llbnRzIGFuZCBvdGhlciBtZXRyaWNzIGNoYW5nZSBmb3IgZWFjaCByZXNhbXBsZSwgYnV0IGJ5IGRlZmF1bHQsIHRoZXNlIHR5cGljYWxseSBhcmVuJ3QgcmV0YWluZWQgYmVjYXVzZSB0aGV5J3JlIGp1c3QgYmVpbmcgdXNlZCB0byBldmFsdWF0ZSBwZXJmb3JtYW5jZS4gIAoKYGBge3Igc2F2ZSBvYmplY3RzfQojIFNhdmUgZm9sZHMsIHByZWRpY3Rpb25zLCBhbmQgcmFuZG9tIGZvcmVzdCByZXN1bHRzCgoKYGBgCgo=